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AFRL  Battlespace  Visualization  Branch 
Display  Characterization  Facility 


Denise  L.  Aleva  and  Frederick  M.  Meyer 
Air  Force  Research  Laboratory 
Steve  Fullenkamp,  Robert  Schwartz  and  Terry  Trissell 
Northrop  Grumman,  Inc. 


INTRODUCTION 


Not  long  ago,  our  only  choices  for  cockpit  displays  were  analog  dials  and  gauges  and  the  occasional  CRT  which  was 
used  primarily  for  imagery  display.  Today,  with  many  new  display  technologies  currently  on-the-shelf  and  others  under 
development,  there  are  many  choices  available  both  for  cockpit  displays  and  for  ground-based  applications.  A  variety  of 
helmet-mounted  and  flat  panel  displays  are  now  available  for  use  in  the  cockpit.  For  ground-based  applications,  large 
screen  displays  allow  greater  amounts  of  information  to  be  displayed  on  a  single  surface  and  shared  among  multiple 
operators. 

Digital  displays  will  play  a  critical  role  in  providing  a  common  battlespace  picture  whether  in  the  aircraft  coclq)it, 
command  and  control  facility  or  carried  by  ground  troops.  Advanced  display  technologies  will  be  key  to  providing  our 
warfighters  with  needed  information. 

Many  new  display  technologies  are  now  on  the  market  or  soon  to  be  matured.  These  displays  offer  higher  resolution, 
greater  dynamic  range  of  luminance,  larger  screen  sizes,  miniaturization  for  helmet/head-mounted  applications,  different 
color  gamuts,  display  flexibility  and  wireless  interfaces.  As  Air  Force  information  display  requirements  become  more 
con^lex,  many  of  these  characteristics  are  expected  to  lend  themselves  to  Air  Force  display  applications. 

The  purpose  of  the  Display  Characterization  Facility  at  WPAFB  is  to  provide  quantitative  performance  data  on  current 
and  upcoming  display  technologies  and  evaluate  these  technologies  for  specific  Air  Force  applications.  This  requires  an 
understanding  not  only  of  the  specific  display  technology  and  its  capabilities  and  limitations  but  also  the  capabilities  and 
limitations  of  the  human  visual  system,  the  tasks  to  be  performed  and  characteristics  of  the  environment  which  may 
affect  the  operator-display  interaction.  To  this  end,  the  Display  Characterization  Laboratory  conducts  both  display 
hardware  measurements  and  assessments  of  human  performance  using  the  displays  under  expected  environmental 
conditions. 

The  Display  Characterization  Facility  is  shown  in  Figure  1.  Facility  activities  include  characterization  of  off-the-shelf 
displays  of  all  types.  In  addition,  the  laboratory  is  often  called  upon  to  evaluate  prototype  displays  which  may  look 
promising  for  Air  Force  applications.  Display  luminance/contrast  capabilities,  color  gamut,  visibility  in  various  ambient 
illumination  conditions,  off  axis  visibility,  reflection  and  night  vision  goggle  con^atibility  are  just  some  of  the  issues 
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Figure  1.  Display  Characterization  Facility  at  Air  Force  Research  Laboratory. 


which  must  be  addressed.  Laboratory  personnel  also  assist  Air  Force  program  managers  and  engineers  in  creating 
display  specifications  for  their  particular  applications. 

As  new  display  technologies  are  developed,  procedures  for  characterization  of  these  new  displays  must  be  standardized. 
Laboratory  personnel  are  working  in  collaboration  with  the  National  Institute  of  Standards  and  Technology  and  the 
Vehicular  Display  Metrology  Working  Group  to  develop  and  standardize  display  characterization  procedures. 

This  paper  will  describe  the  display  measurements  most  often  performed  in  our  laboratory  and  discuss  why  these 
measurements  are  inqjortant. 


MAJOR  EQUIPMENT 


The  laboratory  contains  state-of-the-art  hardware  and  software  for  both  in-laboratory  ad  on¬ 
site  display  characterization.  Some  of  the  major  items  of  equipment  are  listed  below: 

•  Instrument  Systems  IS-320  Spectroradiometer  (2) 

•  Microvision  SS-330  Display  Analysis  System 

•  Eldim  EZ-Contrast  Measurement  System 

•  Hoffman  LS-65  Calibrated  Light  Source 

•  Hoffinan  LM-33-52  Sunlight  Readability  Test  Set 

•  Photo  Research  PR-650  Portable  Spectroradiometer 

•  Rooftop  Dome  Observatory  (Figure  2) 


COMMON  DISPLAY  MEASUREMENTS 


Figure  2.  Dome 
Observatory 


While  the  selection  of  measurements  to  be  performed  on  any  display  will  depend  on  where 

and  how  the  display  is  intended  to  be  usei  there  are  a  number  of  measurements  which  are  applicable  to  almost  any 
display.  These  in  include: 


•  Display  Luminance  Range  and  Contrast  Ratio 

•  Gray  Scale  and  Display  Gamma 

•  Display  Color  Gamut  &  Color  Coordinates 

•  Shadowing 

•  Uniformity  of  color  and  luminance 

•  Viewing  Angle  Effects  upon  Luminance,  Contrast  and  Color 

•  Ambient  Illumination  Effects  upon  Contrast 

•  Checkerboard  Contrast 

•  Response  Time 

•  Power  Consumption 

In  addition,  there  are  two  other  measurements  which  are  often  very  important  for  military  applications.  These  are: 

•  Sunlight  Readability 

•  Night  Vision  Compatibility 

Each  of  these  measurements  will  be  discussed  in  further  detail  in  the  following  paragraphs. 

Display  and  measurement  equipment  set  up  is  critical  to  any  display  evaluation.  All  measurements  are  conducted  in 
accordance  with  applicable  military  standards  and  the  Video  Electronics  Standards  Association  (VESA)  display 
measurement  standards.  The  calibration  guidelines  and  set  up  procedures  are  set  forth  in  the  VESA  Flat  Panel  Display 
Measurements  Standard,  Version  2.0  Publication.  In  house  calibration  facilities  assure  that  all  measurements  are 
traceable  to  National  Institute  of  Standards  and  technology  (NIST)  standards, 


Display  Luminance  Range  and  Contrast  Ratio 

Display  luminance  range  and  contrast  ratio  are  determined  by  measuring  the  luminance  of  full  screen  white  and  full 
screen  black.  These  two  measurements  are  typically  made  in  the  center  of  the  display.  The  units  are  typically  expressed 
in  candelas  per  meter  squared  (cd/m^).  The  ftill  screen  white  measurement  requires  maximum  output  fi-om  each  of  the 
primary  display  elements,  Red,  Green  and  Blue.  For  a  display  system  with  8  bit  color  depth  the  inputs  would  be:  red  = 
255,  green  =  255,  blue  =  255.  For  systems  with  more  color  depth  the  resulting  inputs  would  be  increased  accordingly  to 
the  maximum  value  possible.  Black  is  acconq)lished  by  setting  the  Red,  Green  and  Blue  to  0,0,0.  The  measurement  of 
the  black  full  screen  is  particularly  susceptible  to  the  effects  of  ambient  lighting  and  room  reflections. 

The  difference  between  the  luminance  of  full  screen  white  and  that  of  full  screen  black  gives  us  the  luminance  range  of 
the  display.  This  is  in^)ortant  if  the  display  is  to  be  used  in  both  day  and  nighttime  conditions.  The  luminance  range 
also  limits  the  number  of  discriminable  gray  shades  that  the  display  can  produce.  This  is  particularly  inqjortant  for 
display  of  continuous  tone  imagery. 

The  full  screen  white  and  black  luminance  values  are  also  used  to  calculate  the  Contrast  Ratio  of  the  Full  Screen.  This 
contrast  ratio  is  expressed  mathematically  by  the  equation:  C  =  Lw/  Lb  where  C  is  the  contrast  ratio,  Lw  is  the  luminance 
of  the  full  screen  white  measurement  and  Lb  is  the  luminance  of  the  full  screen  black  measurement.  Display  contrast  is 
in:q)ortant  for  legibility,  particularly  of  high  spatial  firequency  information  such  as  alphanumerics. 


Grey  Scale  and  Display  Gamma 

Eight  or  more  full  screen  grey  values  are  displayed  and  their  luminance  measured.  The  input  values  for  the  grey  levels 
are  equally  spaced  between  the  input  value  that  displays  the  minimum  luminance  (black,  zero)  and  the  input  value  that 
displays  the  maximum  luminance  (white,  maximum).  These  measurements  allow  the  relationship  between  input  value 
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and  output  luminance  to  be  established.  This  relationship  is  called  the  gamma  of  the  display.  Also  shown  in  the  figure 
are  the  effects  of  filtering  the  display  with  either  a  night  vision  con^atibility  filter  or  a  neutral  density  filter. 
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Gray  scale  bit  values 

Figure  3.  Gamma  of  a  laptop  computer  liquid  crystal  display  (LCD). 
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At  low  luminances,  small  changes  in  luminance  are  detectable  by  the  human  eye.  However,  as  luminance  increases, 
larger  steps  in  luminance  are  required  in  order  for  the  eye  to  discern  the  differences.  Therefore,  for  most  applications,  it 
is  desirable  that  the  display  gamma  increase  by  small  steps  at  low  luminances  and  by  increasingly  larger  steps  as 
luminance  increases. 


Display  Color  Gamut 

The  color  gamut  of  a  display  describes  in  CIE  space  the  range  of  colors  that  the  display  can  produce.  Figure  4a  shows 
CIE  1976  space.  It  illustrates  the  shape  of  CIE  1976  space  and  the  color  produced  at  any  x,y  point  within  the  space. 
This  is  essentially  the  range  of  colors  that  a  human  can  perceive. 

Three  color  measurements  are  necessary  to  determine  a  display’s  gamut.  Red  is  measured  at  maximum  output  with 
Green  and  Blue  set  to  zero.  That  gives  the  triangle  vertex  in  the  red  in  Figure  2b.  Then  Green  is  measured  at  maximum 
output  with  Red  and  Blue  set  to  zero.  That  gives  the  triangle  vertex  in  the  green.  Finally,  Blue  is  measured  at  maximum 
output  with  Red  and  Green  set  to  zero.  That  gives  the  triangle  vertex  in  the  blue.  Straight  lines  are  drawn  between 
vertices  (Green,  Red),  (Green,  Blue)  and  (Blue,  Red)  to  form  the  triangle  that  defines  the  display  color  gamut.  Figure  4b 
illustrates  the  result  of  a  gamut  measurement  on  an  example  monitor.  The  monitor  can  produce  only  the  range  of  colors 
shown  in  the  color  triangle,  a  subset  of  what  a  human  can  perceive. 
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The  color  of  the  display  white  and  black  is  generally  measured  along  with  the  color  gamut.  The  color  of  the  white  and 
black  screen  is  reported  as  an  x,y  pair  in  CIE  space. 


Figure  4a.  Full  spectrum  of  color  visible  Figure  4b.  Measured  color  gamut  of  a 

to  the  human  eye.  display 


Uniformity  of  Color  and  Luminance 

Uniformity  of  color  and  luminance  refers  to  the  variability  that  exists  across 
the  area  of  a  full-screen  display  when,  in  theory,  different  areas  of  the 
display  should  have  identical  color  and  luminance  characteristics. 

Measurements  are  made  at  specific  points  on  the  full  screen  white  display  to 
determine  how  much  luminance  and  color  variation  is  present  in  a  full 
screen  white  display.  The  display  format  shown  in  Figure  5  is  used  as  an 
alignment  guide.  Typically,  either  five  (four  comers  and  center)  or  nine 
(four  comers,  center,  and  midpoint  of  top,  bottom  and  each  side  of  the 
display)  points  are  measured.  The  deviations  for  luminance  are  reported  as 
a  percentage  difference  from  the  maximum  white  measured.  Color 
differences  are  reported  in  terms  of  u’  v’  differences.  The  same  procedure 
is  followed  for  each  of  the  red,  green,  and  blue  display  primaries. 

Full  screen  black  is  measured  at  the  same  points,  thus  giving  not  only  the 
uniformity  of  the  black  screen  but  also  enabling  us  to  conpute  contrast  ratio 
at  each  point  and  evaluate  the  uniformity  of  contrast  ratio. 

Figure  5.  Alignment  guide  for  display 

Color  and  luminance  uniformity  contribute  to  making  the  display  uniformity  measurements, 
aesthetically  pleasing.  However,  it  is  also  very  inportant  if  the  display 

format  enploys  color  or  brightness  coding.  A  red  symbol  should  look  the  same  no  matter  where  on  the  display  it 
appears. 
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Viewing  Angle  Effects  upon  Color  and  Contrast 


Viewing  angle  measurements  are  made  to  assess  what  happens  when  a  display  is  viewed  at  angles  other  than  the 
standard  straight  on  perpendicular  viewing  condition.  The  same  area  of  the  display  is  measured,  while  the  angle  between 
the  measuring  device  and  the  display  is  changed.  Typically,  the  center  of  the  display  is  measured.  Either  the  display  or 
the  measuring  device  is  rotated  and  repositioned  to  assure  that  the  same  area  of  the  screen  is  being  measured.  It  is 
inportant  to  measure  the  same  area  of  the  display  to  assure  that  spatial  inhomogeneity  does  not  contaminate  the 
measurements.  White  luminance,  black  luminance  and  chromaticity  coordinates  are  measured  to  assess  the  changes  in 
luminance,  contrast  ratio  or  color  that  may  come  about  as  an  effect  of  viewing  angle.  Viewing  angles  of  +/-  30  degrees 
horizontal  and  +/-  15  degrees  vertical  are  generally  measured  as  a  check  of  a  manufacturer’s  viewing  angle  performance 
specification;  however  the  selection  of  what  viewing  angle  to  assess  can  be  driven  by  a  particular  application.  For  some 
displays,  contrast  will  decrease  significantly  with  increased  viewing  angle.  Changes  in  color  may  also  occur  with 
increased  viewing  angle. 


Ambient  Illumination  Effects  upon  Contrast 

This  is  simply  a  matter  of  measuring  display  contrast  in  the  same  lighting  environment  in  which  the  display  will  be  used. 
Ambient  illumination  which  strikes  the  surface  of  the  display  may  be  reflected  back  to  the  viewer’s  eye,  thus  reducing 
the  perceived  contrast  of  the  display. 


Checkerboard  Luminance  and  Contrast 


Displaying  a  checkerboard  of  white  and  black  squares  on  the  display  and 
measuring  the  luminance  of  the  squares  gives  us  luminance  and  contrast 
data  for  the  display  as  it  would  typically  be  used  (i.e.,  with  light  and  dark 
areas  rather  than  all  white  or  all  black).  A  checkerboard  pattern  is 
presented  on  the  display  that  is  being  measured.  The  size  of  the 
individual  squares  is  determined  by  the  size  of  the  area  that  the  light 
measuring  device  samples.  The  500  pixel  minimum  should  be 
maintained  during  this  measurement.  The  measurement  area  of  the 
measuring  device  should  fit  easily  within  a  white  or  black  square  to 
assure  that  only  the  desired  area  is  being  measured.  A  cone-shaped 
device,  called  a  fustrum,  (shown  in  Figure  6)  is  used  to  block  out  any 
stray  light  firom  the  surroimding  display  area.  Measurements  are  made  in 
the  center  of  the  display  for  both  the  white  and  black  squares.  This  is 
made  easy  by  creating  a  reverse  pattern  and  displaying  one  of  the  patterns 
to  measure  die  white  square  and  then  displaying  die  other  pattern,  which 
is  the  “negative”  of  the  first,  to  measure  the  black  square.  If  this  is  not 
possible,  it  is  permissible  to  move  the  display  or  monitor  to  measure  the 
squares,  while  remaining  as  close  as  possible  to  the  center  of  the  display. 
Checkerboard  patterns  should  contain  an  equal  number  of  black  and 
white  squares.  Additional  measmements  at  other  positions  on  the  screen 
using  the  checkerboard  pattern  will  allow  the  consistency  of  contrast 
levels  produced  at  these  positions  to  be  evaluated.  The  luminance  is 
recorded  and  the  contrast  is  confuted  in  the  same  manner  as  it  was  done 
for  full  screen  white  and  black. 


Figure  6.  Checkerboard  contrast 
measurement  set-up  using  fustrum  to 
block  stray  light. 


Response  Time 

Pixel  response  time  measures  how  quickly  a  pixel  can  turn  off  or  turn  on.  Slow  pixel  response  times  result  in  a  blurring 
of  motion  sequences.  Response  time  measurements  require  a  measurement  device  with  linear  temporal  characteristics 
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and  a  response  time  at  least  ten  times  shorter  than  the  expected  pixel  response  time  of  the  display  being  measured. 
Typically,  the  output  of  the  measuring  device  is  connected  to  a  storage  oscilloscope  for  evaluation.  The  exact  level  at 
which  a  pixel  is  considered  to  be  “on”  or  “off’  is  somewhat  subjective  and  application  driven.  The  shape  of  the 
excitation/extinction  curve  drives  this  criterion.  When  changing  from  black  to  white,  90  percent  is  often  considered  full 
on.  When  changing  from  white  to  black  10  percent  is  often  considered  full  off.  Remember  that  this  measurement 
assesses  the  speed  of  the  entire  system  (conputer,  drive  electronics,  etc.)  and  not  just  the  response  of  the  display 
elements.  Also  remember  that  the  shape  of  the  “pixel  on”  portion  of  the  response  curve  is  not  just  a  mirror  image  of  the 
“pixel  off’  portion  of  the  response  curve.  This  means  that  both  on  and  off  portions  of  the  response  curve  must  be 
measured  and  not  extrapolated  one  from  the  other.  Figure  7  shows  example  plots  of  typical  pixel  rise  and  fall  times. 
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Figure  10.  Exanq)le  plots  of  pixel  rise  (left)  and  fall  (right)  times. 


Power  Consumption 

Power  consumption  measurements  are  useful  when  displays  are  used  in  remote  locations  or  where  a  limited  amount  of 
electricity  is  available  to  power  the  required  equipment.  Power  consumption  measurements  should  be  made  for  the 
worst  case,  where  the  display  is  adjusted  in  a  manner  that  the  maximum  possible  power  consumption  occurs.  In  cases 
where  this  is  atypical  and  where  typical  settings  are  available  and  achievable,  power  consumption  measurements  can  be 
made  at  those  settings  or  conditions. 


Sunlight  Readability 

The  evaluation  of  a  display  panel  to  determine  its  hi-ambient  legibility  involves  evalixating  how  well  it  maintains  good 
contrast  in  lighting  conditions  typical  for  where  the  display  will  be  used.  The  human  eye  adjusts  its  aperture  (pupil)  size 
depending  on  the  ambient  lighting.  This  can  restrict  the  amount  of  light  from  a  display.  If  the  display  luminance  is  not 
high  enough  and  the  display  contrast  is  too  low  the  images  on  the  display  will  not  be  legible,  at  least  not  rapidly  legible. 

Figure  1 1  illustrates  the  effects  of  direct  sunlight.  The  critical  requirement  for  good  display  legibility  in  HI- AMBIENT 
(sunlight)  is  a  combination  of  screen  reflectivity  and  luminance  level  where  both  inq)act  contrast. 
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MIL-HDBK-87213:  Provides  guidance  to  assure  display  legibility  in  the  most  critical  ambient  illumination  conditions. 
A  Hoffman  LM-33-52  sunlight  contrast  measurement  system  is  used  for  this  test.  The  setup  consists  of  the  following:  a 
107  klx  (10  kfc)  direct  illumination  source  (projection  source)  normal  to  the  center  of  the  screen,  a  diffuse  source  of 
7160  cd/m^  (2055  fL)  at  30“  from  the  right  of  the  screen,  and  the  photometer  30“  to  die  left  of  the  screen. 


SAE:  J1757  Vehicular  Display  Metrology:  Provides  guidance  using  typical  laboratory  optical  measuring  equipment  to 
configure  representative  light  sources.  The  objective  is  to  simulate  the  necessary  conditions  to  determine  the  display  s 
contrast  conpliance  for  minimum  contrast  within  a  viewing  cone  for  worst  case  ambient  conditions. 


Figure  1 1 .  Display  in  normal  office  lighting  (left)  and  in  direct  sunlight  (right) 


Night  Vision  Imaging  System  (NVIS)  Compatibility 

With  inrrpaging  usc  of  Night  Vision  Goggles  (NVGs)  in  aircraft  cockpits,  care  must  be  taken  to  assure  that  cockpit 
displays  do  not  interfere  wifli  the  NVGs.  NVGs  are  particularly  sensitive  to  wavelengths  greater  than  570  nm.  MIL- 
STD-3009  defines  the  radiant  energy  interface  requirements  and  test  procedures  applicable  to  NVIS  compatible  lighting 
systems  for  new  or  modified  aircraft  lighting  equipment  and  crew  stations.  To  determine  NVIS  con^atibility,  display 
radiance  ouqjut  as  a  fimction  of  wavelength  is  measured  using  a  spectral  scanning  radiometer.  NVIS  conqjatibility  is 
calculated  from  the  spectral  scan  using  the  radiometer’s  software.  Figure  12  shows  spectral  scans  of  a  laptop  conqjuter 
display  with  and  without  two  filters  -  a  neutral  density  filter  and  an  NVIS  compatible  filter. 


Figure  12.  Spectral  scans  of  a  laptop  con^uter  display  with  and  without  a  neutral  density  filter  (left  -  not  NVIS 
conqiatible)  and  with  NVIS  con5)atible  filter  (right). 
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Figure  13  illustrates  the  effects  of  the  non  filtered  and  filtered  display  upon  NVGs.  The  unage  at  the  left  shows  the  view 
through  an  NVG  of  a  resolution  chart  with  an  NVIS  compatible  filtered  display.  The  image  on  the  right  is  the  same  view 
with  the  filter  removed  fi-om  the  display.  The  radiant  energy  from  the  unfiltered  display  causes  the  automatic  gain 
control  of  the  NVG  to  decrease  the  NVG  gain,  thus  leaving  the  resolution  chart  invisible. 


Figure  13.  View  through  NVG  of  a  paper  resolution  chart  behind  a  display  with  NVIS  conqiatible  filter  (left) 
and  same  view  with  filter  removed  from  display  (right). 


SUMMARY 

The  Air  Force  Research  Laboratory’s  Display  Characterization  Facility  supports  a  variety  of  research  projects  in  the 
areas  of  display  hardware  development,  human  visual  perception  and  information  presentation.  The  objective  of  this 
research  is  to  maximize  operator  acquisition  of  visual  information  and  facilitate  decision-making  through  optimization  of 
display  parameters  and  design  of  display  formats.  A  systems-engineering  approach  is  required,  taking  into  account  the 
display  performance  parameters,  operator  visual  system  capabilities  and  limitations  as  well  as  information  processing 
strategies  and  capabilities  and  environmental  factors.  Reliable  objective  measures  of  display  performance  parameters  are 
a  critical  first  step  in  optimizing  operator  performance. 


Information  presentation  technologies  available  now  and  in  the  near  future  will  have  the  potential  for  supporting 
battlespace  visualization  with  multisensory  information  interfaces  and  virtual  collaboration  environments.  However, 
human  factors  guidelines  will  be  needed  for  determining  the  combination  of  information  presentation  technologies  most 
appropriate  for  a  particular  operational  environment  and  optimizing  information  presentation. 
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